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DETAILED ACTION 

1 . The response received on 24 October 2007 lias been placed in tlie file and was 
considered by the examiner. An action on the merits follows. 

Response to Amendment 

2. The applicant's amendments filed on 18 October 2007 have been fully 
considered. 

Response to Arguments 

3. Applicant's arguments with respect to claims 1 -2, 4-19, and 21 -22 have been 
considered but are moot in view of the new ground(s) of rejection. 

Claim Rejections - 35 USC § 101 

4. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, macliine. manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

5. Claim 21 is rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter. Claim 21 is drawn to functional descriptive 
material NOT claimed as residing on a computer readable medium. MPEP 
2106.IV.B.1(a) (Functional Descriptive Material) states: 

"Data structures not claimed as embodied in a computer-readable medium are 
descriptive material per se and are not statutory because they are not capable of 
causing functional change in the computer." 
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"Such claimed data structures do not define any structural or functional 
interrelationships between the data structure and other claimed aspects of the invention 
which permit the data structure's functionality to be realized." 



Claim 21, while defining a computer program product, does not define a 
"computer-readable medium" and is thus non-statutory for that reasons. A computer 
program product can range from paper on which the program is written, to a program 
simply contemplated and memorized by a person. The examiner suggests amending 
the claim to embody the program on "computer-readable medium" in order to make the 
claim statutory. 

"In contrast, a claimed computer-readable medium encoded with the data 
structure defines structural and functional interrelationships between the data structure 
and the computer software and hardware components which permit the data structure's 
functionality to be realized, and is thus statutory." - MPEP 2106.IV.B.1(a) 



The examiner suggests that 35 USC § 101 can be resolved if the applicant 
replaces "a computer program product" with "a computer readable medium." 



Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, If the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary sl<ill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 
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7. Claims 1-2, 5-7, and 18-22 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Satoh et al (EP 111 7074 A2) in view of Yonezawa et al. (US 
2002/0075286). 

Regarding claim 1 , Satoli discloses an image generating system (abstract; Figs. 
2, 4, 14, and 17), comprising: a database (Fig. 1-104, "AR Game Apparatus" and Fig. 7) 
which stores (Fig. 7-703, :RAM"; paragraph [0093]) first shape data which represents a 
three dimensional shape of a first area including at least a part of an object area in a 
real world (Fig. 1-105; paragraphs [0037] and [0038], "real object"); a camera which 
shoots a second area including at least a part of the object area (Fig. 2-103, Fig. 4-103, 
Fig. 14-103, and Fig. 17-103; paragraphs [0036]-[0037], [0046], and [0100], "actually 
sensed video") and an image generating apparatus which generates an image of the 
object area using a picture shot by the camera and the first shape data (Fig. 2-104, Fig. 
4-104, Fig. 14-104, and Fig. 17-104; paragraph [0037], "AR game apparatus") , wherein 
said image generating apparatus includes: a data acquiring unit which acquires the first 
shape data from said database (Fig. 2-201, Fig. 4-201, Fig. 14-201, and Fig. 17-201; 
paragraph [0063], "game state manager"); a picture acquiring unit which acquires the 
picture from said camera (FIG. 8-S803; paragraphs [0036], [0046], and [0071]); a first 
generating unit which generates an image of the first area by setting a predetermined 
viewpoint and a view direction and rendering the first shape data (Fig. 2-202 and 212, 
Fig. 4-202 and 212. Fig. 14-202 and 212. and Fig. 17-202 and 212; paragraphs [0042], 
[0045], and [0048], "Objective viewpoint video generator" and "Subjective viewpoint 
video generator"); a second generating unit which generates an image of the second 
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area when viewed from tlie viewpoint toward the view direction by using the picture 
(paragraphs [0044] and [0046]); and a compositing unit which composites the image of 
the first area with the image of the second area to generate the image of the object area 
(Fig. 2-203, Fig. 4-203, Fig. 14-203, and Fig. 17-203; paragraphs [0046] and [0084]; 
claim 1 "Objective viewpoint video composition unit"). However, Satoh does not 
disclose the last claim limitation of claim 1 . Instead of Satoh, Yonezawa does. 
Yonezawa discloses compositing unit generates the image of the object area by 
complementing an area that is not shot by the camera with the image of the first area 
generated from the first shape data (paragraphs [0046]-[0047] and [0049]-[0050]). 

It would have been prima facie obvious to one of ordinary skill in the art at the 
time the invention was made to use Yonezawa's invention in Satoh's invention to 
provide a more real composite image as suggested by Yonezawa (paragraph [0006]). 

Regarding claim 2, Satoh further discloses an image generating system 
(abstract; Figs.2, 4, 14, and 17) wherein: said image generating system includes a 
plurality of cameras located at a plurality of positions (Fig. 9; paragraphs [0096]-[0102]); 
said image generating apparatus further comprises a calculating unit which calculates 
second shape data which represents a three dimensional shape of the second area 
using a plurality of the pictures acquired from said plurality of cameras (paragraphs 
[0056]-[0058]); said second generating unit sets the viewpoint and the view direction 
and renders the second shape data to generate the image of the second area 
(paragraphs [0035], [0044] and [0046]). 
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Regarding claim 5, Satoli further discloses an image generating system 
(abstract; Figs.2, 4, 14, and 17) wherein said database stores the first shape data 
obtained by modeling an area which does not change in a short terni in the object area 
(paragraphs [0007]-[0009], [0042]-[0043]. [0045], and [0063]). 

Regarding claim 6, Satoh further discloses an image generating system 
(abstract; Figs.2, 4, 14, and 17) wherein said database stores the first shape data 
obtained by modeling an area which does not change in a short temn in the object area 
(paragraphs [0007]-[0009], [0042]-[0043], [0045], and [0063]). 

Regarding claim 7, Satoh further discloses an image generating system 
(abstract; Flgs.2, 4, 14, and 17) wherein said database stores the first shape data 
obtained by modeling an area which does not change in a short term in the object area 
(paragraphs [0007]-[0009], [0042]-[0043], [0045], and [0063]). 

Regarding claim 18, Satoh discloses an image generating apparatus (abstract; 
Figs. 2, 4, 14, and 17), comprising: a data acquiring unit which acquires first shape data 
which represents a three dimensional shape of a first area including at least one part of 
an object area in a real world (Fig. 1-105; paragraphs [0037] and [0038], "real object") 
from a database (Fig. 2-201, Fig. 4-201, Fig. 14-201, and Fig. 17-201; paragraph 
[0063], "game state manager") which stores the first shape data (paragraphs [0035] and 
[0063], "virtual objects"); a picture acquiring unit which acquires a picture of a second 
area including at least one part of the object area shot (Fig. 2-103, Fig. 4-103, Fig. 14- 
103, and Fig. 17-103; paragraphs [0036]-[0037], [0046], and [0100], "actually sensed 
video") by a plurality of cameras located at a plurality of positions from the cameras 



Application/Control Number: Page 7 

10/780.303 

Art Unit: 2624 

(Fig. 9; paragraphs [0096H0102]); a first generating unit which generates an image of 
the first area by setting a predetermined viewpoint and a view direction and rendering 
the first shape data (Fig. 2-202 and 212, Fig. 4-202 and 212, Fig. 14-202 and 212, and 
Fig. 17-202 and 212; paragraphs [0042], [0045], and [0048], "Objective viewpoint video 
generator" and "Subjective viewpoint video generator"); a second generating unit which 
generates an image of the second area when viewed from the viewpoint toward the 
view direction by using the picture shot (paragraphs [0044] and [0046]); and a 
compositing unit which composites the image of the first area with the image of the 
second area to generate the image of the object area (Fig. 2-203, Fig. 4-203, Fig. 14- 
203, and Fig. 17-203; paragraphs [0046] and [0084]; claim 1 "Objective viewpoint video 
composition unit"). However, Satoh does not disclose the last claim limitation of claim 
18. Instead of Satoh, Yonezawa does. Yonezawa discloses compositing unit generates 
the image of the object area by complementing an area that is not shot by the camera 
with the image of the first area generated from the first shape data (paragraphs [0046]- 
[0047] and [0049]-[0050]). 

It would have been prima facie obvious to one of ordinary skill In the art at the 
time the Invention was made to use Yonezawa's Invention In Satoh's Invention to 
provide a more real composite image as suggested by Yonezawa (paragraph [0006]). 

Regarding claim 19, claim 19 is analogous and conresponds to claim 18. See 
rejection of claim 18 for further explanation. 
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Regarding claim 21 , Satoli discloses a program executable by a computer 
(paragraphs [0061]-[0062]; claims 19-27). Moreover, claim 21 is analogous and 
conresponds to claim 18. See rejection of claim 18 for further explanation. 

Regarding claim 22, Satoh discloses a computer-readable recording medium 
which stores a program executable by a computer (paragraphs [0061]-[0063]; claims 
19-27). Moreover, claim 22 is analogous and corresponds to claim 18. See rejection of 
claim 18 for further explanation. 

8. Claims 4 and 8 are rejected under 35 U.S.C. 1 03(a) as being unpatentable over 
Satoh et al (EP 1 1 1 7074 A2) in view of Yonezawa et al. (US 2002/0075286). and further 
in view of Anabuki et al. (US 6,633,304). 

Regarding claim 4, Satoh and Yonezawa disclose all the previous claim 
limitations except the claim limitation specified in claim 4. However, Anabuki discloses 
second generating unit renders the area which is not represented by the second shape 
data with a transparent color when rendering the second shape data (col.7, lines 19-25); 
said compositing unit generates the image of the object area by ovenft^riting the image of 
the second area with the image of the first area (col. 2, lines 16-24). 

It would have been prima facie obvious to one of ordinary skill in the art at the 
time the invention was made to use Yonezawa's Invention and Anabuki's invention in 
Satoh's invention to provide a mixed reality presentation apparatus and control method 
suggested by Anabuki (col. 2, lines 16-20). 

Regarding claim 8, Satoh further discloses an image generating system 
(abstract; Figs.2, 4. 14, and 17) wherein said database stores the first shape data 
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obtained by modeling an area which does not change in a short temi in the object area 
(paragraphs [0007]-[0009], [0042]-[0043], [0045], and [0063]). 
9. Claims 9-1 1 , and 1 3-1 5 are rejected under 35 U.S.C. 1 03(a) as being 
unpatentable over Satoh et al (EP 1 1 1 7074 A2) in view of Yonezawa et al. (US 
2002/0075286), and further in view of Ikeuchi ("Modeling from Reality"). 

Regarding claim 9, Satoh and Yonezawa disclose all the previous claim 
limitations except the claim limitation specified in claim 9. However, Ikeuchi teaches a 
database stores first color data which represents a color of the first area (section 3); 
said image generating apparatus further includes a lighting calculating unit which 
calculates a situation of a lighting in the picture shot by comparing the first color data 
acquired from said database with color data of the picture shot (sections 3.1, 3.2, 4, 4.1 , 
4.2, and 4.3). 

It would have been prima facie obvious to one of ordinary skill in the art at the 
time the invention was made to use Yonezawa's invention and Ikeuchi's method in 
Satoh's invention to effectively use for synthesizing realistic object images as suggested 
by Ikeuchi (section 6). 

Regarding claim 10, Ikeuchi further teaches a database stores first color data 
which represents a color of the first area (section 3); said image generating apparatus 
further includes a lighting calculating unit which calculates a situation of a lighting in the 
picture shot by comparing the first color data acquired from said database with color 
data of the picture shot (sections 3.1 , 3.2, 4, 4.1, 4.2, and 4.3). 
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Regarding claim 1 1 , ll<euchi further teaclies a database stores first color data 
wtiicii represents a color of the first area (section 3); said image generating apparatus 
further Includes a lighting calculating unit which calculates a situation of a lighting in the 
picture shot by comparing the first color data acquired from said database with color 
data of the picture shot (sections 3.1 , 3.2, 4, 4.1 , 4.2, and 4.3). 

Regarding claim 13, Ikeuchi further teaches a database stores first color data 
which represents a color of the first area (section 3); said image generating apparatus 
further includes a lighting calculating unit which calculates a situation of a lighting in the 
picture shot by comparing the first color data acquired from said database with color 
data of the picture shot (sections 3.1, 3.2, 4, 4.1, 4.2, and 4.3). 

Regarding claim 14, Ikeuchi further teaches first generating unit adds an effect of 
lighting similar to the lighting in the picture shot to the image of the first area in 
consideration of the situation of the lighting (sections 3.1, 3.2, 4, 4.1, 4.2, and 4.3). 

Regarding claim 15, Ikeuchi further teaches first generating unit adds a 
predetermined effect of lighting to the image of the first area (sections 3.1 , 3.2, 4, 4.1 , 
4.2, and 4.3); said second generating unit adds the predetermined effect of lighting to 
the image of the second area, after once removing the effect of lighting from the image 
of the second area (sections 3.1 , 3.2, 4, 4.1 , 4.2, and 4.3). 

10. Claim 12 is rejected under 35 U.S.C. 103(a) as being unpatentable over Satoh et 
al (EP 1 1 17074 A2) in view of Yonezawa et al. (US 2002/0075286), and further in view 
of Anabuki et al. (US 6,633,304) and Ikeuchi ("Modeling from Reality"). 
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Regarding claim 12, Satoh, Yonezawa, and Anabul<i disclose all the previous 
claim limitations except the claim limitation specified in claim 12. However, Ikeuchi 
teaches a database stores first color data which represents a color of the first area 
(section 3); said image generating apparatus further includes a lighting calculating unit 
which calculates a situation of a lighting in the picture shot by comparing the first color 
data acquired from said database with color data of the picture shot (sections 3.1 , 3.2, 
4, 4.1,4.2, and 4.3). 

It would have been prima facie obvious to one of ordinary skill in the art at the 
time the invention was made to use Yonezawa's invention, Anabuki's invention, and 
Ikeuchi's method in Satoh's invention to effectively use for synthesizing realistic object 
images as suggested by Ikeuchi (section 6). 

11. Claims 16 and 17 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Satoh et al (EP 1 1 1 7074 A2) in view of Yonezawa et al. (US 2002/0075286), and 
further in view of Kondo (US 6,812,924) and Sawada (US 5,844,625). 

Regarding claim 16, Satoh and Yonezawa disclose all the previous claim 
limitations, including the image generating system (abstract; Figs. 2, 4, 14, and 17), a 
database storing a plurality of the first shape data corresponding to the object areas of a 
plurality of times (paragraph [0063]), and the image generating apparatus (Fig. 2-104, 
Fig. 4-104, Fig. 14-104. and Fig. 17-104; paragraph [0037], "AR game apparatus") 
disclosed by Satoh, except the other claim limitations of claim 16. However, Kondo 
discloses a first selecting unit which selects the first shape data to be acquired by the 
data acquiring unit among the plurality of the first shape data stored in said database 



Application/Control Number: Page 12 

10/780,303 

Art Unit: 2624 

(Figs. 24 and 25; dol. 10, lines 18-26; col. 11, lines 20-29; claim 7). Sawada discloses 
storing the picture shot(col. 2, lines 43-49, col. 5, lines 28-37) and a second selecting 
unit which selects the picture shot to be acquired by the picture acquiring unit among 
the pictures stored in said recording apparatus (col. 5, lines 64-67; col. 6, lines 1-6). 

It would have been prima facie obvious to one of ordinary skill in the art at the 
time the invention was made to use Yonezawa's invention, Kondo's invention, and 
Sawada's invention in Satoh's invention to provide a an apparatus for obtaining shape 
data (Kondo, col. 1, lines 48-50) and a picture processing apparatus (Sawada, col. 2, 
lines 20-21). 

Regarding claim 17, Yonezawa further discloses second selecting unit selects 
the first shape data corresponding to the time when the picture selected by said first 
selecting unit was shot (paragraphs [0098H0099], "synchronization"). 

Conclusion 

12. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 . 1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 



Application/Control Number: 
10/780.303 

Art Unit: 2624 • 



Page 13 



the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any Inquiry concerning this communication or earlier communications from the 
examiner should be directed to John Wahnkyo Lee whose telephone number is (571 ) 
272-9554. The examiner can nonnally be reached on Monday - Friday (Alt.) 7:30 a.m. - 
5:00 p.m.. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Jingge Wu can be reached on (571) 272-7429. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Infomiation regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status infomiation for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



John W. Lee 

(AU 2624) 




